Skip to content

Validate activity task stamps before reuse #8536

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Nov 4, 2025
Merged

Validate activity task stamps before reuse #8536

merged 6 commits into from
Nov 4, 2025

Conversation

@yiminc
Copy link
Member

@yiminc yiminc commented Oct 24, 2025

Summary

  • add the activity task stamp to IsActivityTaskValidRequest and generated API stubs
  • require history's activity validation logic to match the stored stamp
  • plumb the stamp from matching task validation and cover the new behavior with tests
  • increment activity stamps for each retry attempt and extend tests for paused retries

Testing

  • GOTOOLCHAIN=local go test -tags test_dep ./service/history/workflow -run TestRetryActivity (fails: go.mod requires go >= 1.25.0 and local toolchain is go1.24.3)

https://chatgpt.com/codex/tasks/task_b_68faffae045c832c9d8220bcbe50bac7

@yiminc yiminc requested review from a team as code owners October 24, 2025 04:57
bergundy
bergundy reviewed Oct 27, 2025
View reviewed changes
service/history/api/isactivitytaskvalid/api.go

ai, ok := mutableState.GetActivityInfo(scheduledEventID)
if ok && ai.StartedEventId == common.EmptyEventID {
if ok && ai.StartedEventId == common.EmptyEventID && ai.GetStamp() == stamp {
Copy link
Member

@bergundy bergundy Oct 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't this fail for existing activity tasks that were generated without a stamp?

Copy link
Member Author

@yiminc yiminc Oct 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stamp was added to AI the same time they were added to the matching tasks. For tasks before that, the Stamp from AI would be 0 and same for the tasks. But I could add a defensive check to enforce only when both stamp are > 0.

Copy link
Member

@bergundy bergundy Oct 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah okay, so this should be fine.

Copy link
Member Author

@yiminc yiminc Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we decided to revert this because the additional check will cause the first task always be valid and that could cause head-of-queue blocking issue on standby side.

bergundy
bergundy reviewed Oct 27, 2025
View reviewed changes
Copy link
Member

@bergundy bergundy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way that this is implemented feels error prone to me. It's too easy to mutate the activity and forget to update the stamp. I skimmed the locations where the activity is updated in the code and I think we may have some edge cases that are not covered (need to double check). I would consider updating the stamp in MutableStateImpl.UpdateActivity for any field change that warrants updating it.

@yiminc
Copy link
Member Author

yiminc commented Oct 29, 2025

The way that this is implemented feels error prone to me. It's too easy to mutate the activity and forget to update the stamp. I skimmed the locations where the activity is updated in the code and I think we may have some edge cases that are not covered (need to double check). I would consider updating the stamp in MutableStateImpl.UpdateActivity for any field change that warrants updating it.

The purpose of this validation is to avoid building up backlog with invalid tasks. There isn't much of negative consequence to mis-update the stamp for any update to activity, as long as the most common case of retry has the stamp updated that would serve the purpose. But if there is better way of doing this, we could improve later.

@bergundy
Copy link
Member

bergundy commented Oct 29, 2025

The way that this is implemented feels error prone to me. It's too easy to mutate the activity and forget to update the stamp. I skimmed the locations where the activity is updated in the code and I think we may have some edge cases that are not covered (need to double check). I would consider updating the stamp in MutableStateImpl.UpdateActivity for any field change that warrants updating it.

The purpose of this validation is to avoid building up backlog with invalid tasks. There isn't much of negative consequence to mis-update the stamp for any update to activity, as long as the most common case of retry has the stamp updated that would serve the purpose. But if there is better way of doing this, we could improve later.

We should do better for when we port this to CHASM.

@yiminc yiminc force-pushed the codex/add-stamp-field-to-isactivitytaskvalid branch from 0fe742b to 65cff5f Compare November 3, 2025 22:59
@yiminc yiminc enabled auto-merge (squash) November 4, 2025 00:26
@yiminc yiminc merged commit 392f51b into main Nov 4, 2025
57 checks passed
@yiminc yiminc deleted the codex/add-stamp-field-to-isactivitytaskvalid branch November 4, 2025 00:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bergundy bergundy bergundy approved these changes

@xwduan xwduan xwduan approved these changes

Assignees

No one assigned

Labels

codex

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@yiminc @bergundy @xwduan @michaely520